Skip to content

Add -files-from option to lint command to avoid ARG_MAX limit#6858

Open
adamrtalbot wants to merge 2 commits intomasterfrom
https/github.com/nextflow-io/nextflow/issues/6857
Open

Add -files-from option to lint command to avoid ARG_MAX limit#6858
adamrtalbot wants to merge 2 commits intomasterfrom
https/github.com/nextflow-io/nextflow/issues/6857

Conversation

@adamrtalbot
Copy link
Collaborator

Summary

  • Adds a -files-from option to nextflow lint that reads file paths from a file (one per line), avoiding the OS ARG_MAX limit when many files are passed as CLI arguments
  • Supports reading from stdin via -files-from -
  • Works additively with positional arguments — both sources are combined

Motivation

When using nextflow lint with tools like pre-commit on large repositories (1700+ .nf files), all filenames are passed as CLI arguments, which can exceed the OS ARG_MAX limit (~2MB on Linux), causing E2BIG: Argument list too long errors.

This is Option B from #6857 — a -files-from flag that reads file paths from a file, following the convention used by rsync, tar, and other Unix tools.

Usage

# Read paths from a file
nextflow lint -files-from filelist.txt

# Read paths from stdin
find . -name '*.nf' | nextflow lint -files-from -

# Combine with positional args
nextflow lint main.nf -files-from extra-files.txt

Changes

  • CmdLint.groovy: Added -files-from parameter and file-reading logic in run() that reads lines, trims whitespace, skips blank lines, and appends paths to args before existing processing
  • CmdLintTest.groovy: Added 3 tests — files-from basic usage, combined with positional args, and empty file list error handling

Testing

All 5 CmdLintTest tests pass (2 existing + 3 new).

Closes #6857

When pre-commit passes many file paths as CLI arguments to
'nextflow lint', the OS ARG_MAX limit can be exceeded, causing
E2BIG errors. Add a -files-from option that reads file paths
from a file (one per line), with support for stdin via '-'.

Closes #6857

Generated by Claude Code

Signed-off-by: adamrtalbot <12817534+adamrtalbot@users.noreply.github.com>
@netlify
Copy link

netlify bot commented Feb 24, 2026

Deploy Preview for nextflow-docs-staging canceled.

Name Link
🔨 Latest commit 0eba08a
🔍 Latest deploy log https://app.netlify.com/projects/nextflow-docs-staging/deploys/69a07c70259f740008603e54

@pditommaso pditommaso force-pushed the master branch 2 times, most recently from d9fa5cd to d752bc2 Compare February 28, 2026 13:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

nextflow lint: support reading file paths from a file or stdin to avoid ARG_MAX limits

1 participant